A Primal/Dual Stump Algorithm for Large Numerical Datasets

We demonstrate a stochastic gradient algorithm that can handle the very large number of stump features generated by considering every possible threshold over numerical, or continuous, features. Our problem is to classify data with continuous features, where small variations in the feature value can result in a different classification decision. Consider for instance packet statistics used for network analysis: features such as packet counts and durations need to be finely thresholded to achieve accurate decisions. To be used as input to vectorial classifiers, these features are often normalized between -1 and +1, or standardized to the unit variance: much of the separation power of the feature is then lost. A more powerful way to build a finely discriminant classifier looks at every possible partition of the input space that can separate training examples, and perform a weighted combination of these partitions. In practice, only coordinate-wise partitions are possible: given feature f and threshold θ, the data is split into two sets {f < θ} and {f ≥ θ}. State-of-the-art algorithms that combine partitions include classification trees and boosted stumps [4]. As a matter of fact, a recent comparative study suggests the best algorithm on continuous data is boosted trees [2]. One explanation is that the input space used by boosted trees or stumps is much larger than the space used by SVMs, and that the traditional kernel trick does not help much. Unfortunately, boosting and classification tree algorithms become very cumbersome to apply on very large datasets, as they require a pass through the entire data for each weak classifier to be added. On the other hand, SVMs have recently been shown to be amenable to online implementations [1, 5]. In this work, we use the SVM paradigm to describe regularized linear classifiers, but one could also apply the Maximum Entropy or the generalized Perceptron paradigms with very similar results. We show that SVM can handle a stump representation as rich as classification trees, and we demonstrate an algorithm that is mostly stochastic. The selection of stumps starts by looking, for each feature, at every possible threshold that partitions the training data in a different way, that is every different value the training data takes for this feature. This implies that each one of the n continuous feature can correspond to up to l stump features, where l is the number of training examples. The dimension of the resulting stump space can reach a nl dimension. This work shows how to deal with this explosive number of features by playing with alternative representations of the data and the weight vector.